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DETAILED ACTION 
Specification 

The disclosure is objected to because of tine following informalities: SMIL is 
repeatedly denoted as SMILE in the instant specification for example (page 8, II. 34, 
page 9, II. 3, and page 10, II. 10, & 25, and page 14, II. 11). Appropriate correction is 
required. 

Drawings 

The disclosure is objected to because of the following informalities: SMIL is 
denoted as SMILE in figure 4, label 44 and figures 6A-C (SMILE source files). 
Appropriate correction is required. 

Response to Arguments 
Applicant's arguments filled 08/07/2009 have been considered but are moot in view of 
the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the phor art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1-29 are rejected under 35 U.S.C. 103(a) as being unpatentable over W3C 

Working Draft 09-November-1997 to Bugaj et al. (Bugaj) (previously cited), in view of 

US Patent 6,442,755 to Lemmons et al. (Lemmons), in view of US Patent 6,615,408 to 

Kaiser et al. (Kaiser). 
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Regarding claim 1, Bugaj teaclnes an SMIL (synchronized media integration 
language) for integrating a set of independent multimedia objects into a synchronized 
multimedia presentation such as slide show synchronized with audio comments or a 
video synchronized with text stream wherein SMIL is an XML-based language (section 
2. II. 1-4, & section 3 II. 1); further Bugaj teaches the general syntax of a SMIL document 
comprising a header and a body wherein both parts contain elements and attributes 
(section 4) which reads on "an XML-based stored in a computer-readable medium for 
encoding a visual cue for visual component of a multimedia presentation, wherein the 
XML-based element is structured for use by a computer to display the multimedia 
presentation including the visual component and the visual cue on a display surface of 
the computer, wherein the XML-based -element comprises". 

Bugaj teaches a layout section of a SMIL document including alternative layout 
elements embedded in a switch element used to determine the placement of the 
presentation (section 5, General Semantics, II. 1-6), however, Bugaj is silent on a visual 
element attribute that defines a visual representation of the visual cue; a spatial element 
attribute that defines spatial characteristics of the visual cue; and a temporal element 
attribute that defines temporal characteristics of the visual cue, wherein the and spatial 
characteristics of the visual cue are defined relative to temporal and spatial 
characteristics of the associated visual component, and wherein the computer 
superimposes a display of the visual associated cue on the display of the computer over 
the visual component in the multimedia presentation using a visual appearance which is 
based on the visual representation of the visual cue as defined in the visual element 
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attribute that defines visual representation of the visual cue, during a period of time 
which is based on the temporal characteristics of the visual cue as defined in the 
temporal element attribute that defines temporal characteristics of the visual cue, and at 
a location over the associated visual element which is based on the spatial 
characteristics of the visual cue as defined in the spatial element attribute that defines 
spatial characteristics of the visual cue. 

The examiner maintains that it was well known in the art to provide "a visual 
element attribute that defines a visual representation of the visual cue; a spatial element 
attribute that defines spatial characteristics of the visual cue; and a temporal element 
attribute that defines temporal characteristics of the visual cue, wherein the and spatial 
characteristics of the visual cue are defined relative to temporal and spatial 
characteristics of the associated visual component, and wherein the computer 
superimposes a display of the visual associated cue on the display of the computer over 
the visual component in the multimedia presentation using a visual appearance which is 
based on the visual representation of the visual cue as defined in the visual element 
attribute that defines visual representation of the visual cue, during a period of time 
which is based on the temporal characteristics of the visual cue as defined in the 
temporal element attribute that defines temporal characteristics of the visual cue, and at 
a location over the associated visual element which is based on the spatial 
characteristics of the visual cue as defined in the spatial element attribute that defines 
spatial characteristics of the visual cue", as taught by Lemmons and Kaiser. 
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In a similar field of endeavor Lemmons teaches a visual element attribute that 
defines a visual representation of the visual cue; a spatial element attribute that defines 
spatial characteristics of the visual cue (col. 9, II. 16-22); and a temporal element 
attribute that defines temporal characteristics of the visual cue (figure 6B (because the 
elements a synchronized temporal element attribute is inherent )), wherein the temporal 
and spatial characteristics of the visual cue are defined relative to temporal and spatial 
characteristics of the associated visual component (col. 9, II. 16-22, & figure 6B), and 
wherein the computer (28) superimposes a display of the visual cue (54) on the display 
(36) of the computer (28) together with the visual component in the multimedia 
presentation (figure 6B), using a visual appearance which is based on the visual 
representation of the visual cue as defined in the visual element attribute that defines 
visual representation of the visual cue, during a period of time which is based on the 
temporal characteristics of the visual cue as defined in the temporal element attribute 
that defines temporal characteristics of the visual cue, and at a location over the 
associated visual element which is based on the spatial characteristics of the visual cue 
as defined in the spatial element attribute that defines spatial characteristics of the 
visual cue (col. 9, II. 16-22, & figure 68 , & col. 8, II. 58-67, & col. 9, II. 1-34, & figure 
7A). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Bugaj by specifically providing "a visual element 
attribute that defines a visual representation of the visual cue; a spatial element attribute 
that defines spatial characteristics of the visual cue; and a temporal element attribute 
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that defines temporal characteristics of the visual cue, wherein the and spatial 
characteristics of the visual cue are defined relative to temporal and spatial 
characteristics of the associated visual component, and wherein the computer 
superimposes a display of the visual associated cue on the display of the computer 
together with the visual component in the multimedia presentation using a visual 
appearance which is based on the visual representation of the visual cue as defined In 
the visual element attribute that defines visual representation of the visual cue, during a 
period of time which is based on the temporal characteristics of the visual cue as 
defined In the temporal element attribute that defines temporal characteristics of the 
visual cue, and at a location over the associated visual element which Is based on the 
spatial characteristics of the visual cue as defined in the spatial element attribute that 
defines spatial characteristics of the visual cue", as taught by Lemmons, for the 
purpose of providing Interactive presentations wherein additional information on the 
presented media is available to user during the presentation. 

Bugaj and Lemmons are silent on wherein the computer superimposes a display 
of the visual cue on the display of the computer over the visual component in the 
multimedia presentation. 

Kaiser teaches wherein the computer superimposes a display of the visual cue 
on the display of the computer over the visual component in the multimedia presentation 
(figure 6D, label 6300, & 6600, & col. 11 , II. 20-26). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to modify the system of Bugaj and Lemmons to include wherein 
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the computer superimposes a display of the visual cue on the display of the computer 
together with the visual component in the multimedia presentation as taught by Kaiser in 
order to allow users to easily recognize the association between the interactive icon and 
the displayed video, thereby increasing the viewer's convenience (col. 1, II. 49-55). 

Regarding Claims 2 and 8, Bugaj teaches schedule elements including begin and 
end times wherein duration is defined as the difference between the end times and 
begin time of an element (section 6.1, General Semantics, II. 3, & section 6.2 page 8, II. 
3) which meets "An XML-based element wherein the temporal characteristics include at 
least two of begin time, end time, and duration". 

Regarding Claims 3 and 9, Bugaj teaches an image element tagged as an XML 
media object element (section 6.4, Syntax) which inherently meets "An XML-based 
element, wherein the visual representation includes color". 

Regarding Claims 4 and 10, Bugaj teaches an Image element (section 6.4, 
Syntax) which meets "An XML-based element, wherein .the visual representation 
includes shape". 

Regarding Claims 5 and 1 1 , Bugaj teaches a layout section of a SMIL document 
including alternative layout elements embedded In a switch element used to determine 
the placement of the presentation (section 5, General Semantics, lines 1-6) which 
meets "An XML-based element, wherein the spatial characteristics include position". 

Regarding Claims 6 and 12, Bugaj teaches a smile document for the newscast 
presentation illustrated in figure 7.1 page 26 wherein the layout and temporal elements 
are controlled by their associated attributes in hierarchal order (section 7.4 page 27) 
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which meets "An XML-based element, wherein the XML-based element for the visual 
cue is nested within an XML- based element that defines the associated visual 
component". 

Regarding claims 7 and 13, Bugaj teaches a news broadcast on the web as 
shown in figure 7.1 to the left and right (section 7.4, page 26, lines 2-9) which meets "In 
an XML-based browser that displays synchronized multimedia presentations on a 
display of a computer to user a method for processing an XML-based element for visual 
cue associated with a visual component of the multimedia presentation comprising" 

In addition Bugaj teaches an XML data structure holding the elements of the of 
the newscast scenario written in XML code comprising elements stored in tags wherein 
each element has its own properly defined attribute including spatial and temporal 
characteristics to perform a presentation (page 27) which meets "storing information 
from the XML-based element concerning the visual component to which the visual cue 
is associated, together with the information from the XML-based element concerning 
visual representation". 

Bugaj teaches a layout section of a SMIL document including alternative layout 
elements embedded in a switch element used to determine the placement of the 
presentation (section 5, General Semantics, lines 1-6), however, Bugaj is silent on 
spatial and temporal characteristics of the visual cue; and in synchronization with 
display of the visual component, displaying the visual cue with the visual representation 
specified, and in the spatial and temporal relationships specified by the spatial and 
temporal characteristics, wherein the defined temporal and spatial characteristics of the 
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visual cue are relative to temporal and spatial characteristics of the associated visual 
component, and wherein the display of the visual is superimposed over the associated 
visual component in the multimedia presentation with a visual appearance based on the 
defined visual representation of the visual cue, during a period of time based on the 
defined temporal characteristics of the visual cue, and at a location over the associated 
visual element based on the defined spatial characteristics of the visual cue. 

The examiner maintains that it was well known in the art to provide "spatial and 
temporal characteristics of the visual cue; and in synchronization with display of the 
visual component, displaying the visual cue with the visual representation specified, and 
in the spatial and temporal relationships specified by the spatial and temporal 
characteristics, wherein the defined temporal and spatial characteristics of the visual 
cue are relative to temporal and spatial characteristics of the associated visual 
component, and wherein the display of the visual is superimposed over the associated 
visual component in the multimedia presentation with a visual appearance based on the 
defined visual representation of the visual cue, during a period of time based on the 
defined temporal characteristics of the visual cue, and at a location over the associated 
visual element based on the defined spatial characteristics of the visual cue", as taught 
by Lemmons and Kaiser. 

In a similar field of endeavor Lemmons teaches a spatial and temporal 
characteristics of the visual cue; and in synchronization with display of the visual 
component, displaying the visual cue with the visual representation specified, and in the 
spatial and temporal relationships specified by the spatial and temporal characteristics. 
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wherein the defined temporal and spatial characteristics of the visual cue are relative to 
temporal and spatial characteristics of the associated visual component (col. 9, II. 16-22, 
& col. 9, II. 16-22, & figure 6B, elements); wherein the visual is superimposed together 
with the associated visual component in the multimedia presentation with a visual 
appearance based on the defined visual representation of the visual cue, during a 
period of time based on the defined temporal characteristics of the visual cue, and at a 
location over the associated visual element based on the defined spatial characteristics 
of the visual cue (col. 9, II. 16-22, & figure 6B , & col. 8, II. 58-67, & col. 9, II. 1-34, & 
figure 7A). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Bugaj by specifically providing "spatial and 
temporal characteristics of the visual cue; and in synchronization with display of the 
visual component, displaying the visual cue with the visual representation specified, and 
in the spatial and temporal relationships specified by the spatial and temporal 
characteristics, wherein the defined temporal and spatial characteristics of the visual 
cue are relative to temporal and spatial characteristics of the associated visual 
component, and wherein the display of the visual is superimposed together with the 
associated visual component in the multimedia presentation with a visual appearance 
based on the defined visual representation of the visual cue, during a period of time 
based on the defined temporal characteristics of the visual cue, and at a location over 
the associated visual element based on the defined spatial characteristics of the visual 
cue", as taught by Lemmons, for the purpose of providing interactive presentations 
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wherein additional information on the presented media is available to user during the 
presentation. 

Bugaj and Lemmons are silent on wherein the display of the visual is 
superimposed over the associated visual component in the multimedia presentation with 
a visual appearance based on the defined visual representation of the visual cue. 

Kaiser teaches wherein the display of the visual is superimposed over the 
associated visual component in the multimedia presentation with a visual appearance 
based on the defined visual representation of the visual cue (figure 6D, label 6300, & 
6600, & col. 11,11.20-26). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to modify the system of Bugaj and Lemmons to include wherein 
the display of the visual is superimposed over the associated visual component in the 
multimedia presentation with a visual appearance based on the defined visual 
representation of the visual cue as taught by Kaiser in order to allow users to easily 
recognize the association between the interactive icon and the displayed video, thereby 
increasing the viewer's convenience (col. 1, II. 49-55). 

Regarding Claim 14, Bugaj teaches schedule elements with begin and end time 
wherein the duration is the difference between the end time and begin time of an 
element (section 6.1, General Semantics, line 3, & section 6.2 page 8, line 3) which 
meets "A computer-readable medium wherein the temporal characteristics include at 
least two o f begin time, end time, and duration". 
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Regarding Claim 15, Bugaj teaclies an image element tagged as an XML media 
object element (section 6.4, Syntax) which inherently meets "A computer-readable 
medium, wherein the visual representation includes color". 

Regarding Claim 16, Bugaj teaches an image attribute (section 6.4, Syntax) 
which meets "A computer-readable medium, wherein the visual representation includes 
shape". 

Regarding Claim 17, Bugaj teaches a layout section of a SMIL document 
including alternative layout elements embedded in a switch element used to determine 
the placement of the presentation (section 5, General Semantics, lines 1-6) which 
meets "A computer-readable medium, wherein the spatial characteristics include 
position". 

Regarding Claim 18, Bugaj teaches a SMIL document for the newscast 
presentation illustrated in figure 7.1 page 26 wherein the layout and temporal elements 
are controlled by their associated attributes in hierarchal order (section 7.4 page 27) 
which meets "A computer-readable medium, wherein the XML-based element for the 
visual cue is nested within an XML- based element that defines the associated visual 
component". 

Regarding claims 22 and 26, Bugaj teaches a news broadcast on the web as 
shown in figure 7.1 to the left and right (section 7.4, page 26, lines 2-9) which meets " A 
method for displaying a synchronized multimedia presentation on a display screen of a 
computer executing an XML-based browser, comprising". 
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In addition Bugaj teaches an XML data structure holding the elements of the of 
the newscast scenario written in XML code comprising elements stored in tags wherein 
each element has its own properly defined attribute including spatial and temporal 
characteristics to perform a presentation (page 27) which meets "receiving XML-based 
data including an XM.L-based element for a visual cue together with an XM'L-based 
element for a visual component contained in the multimedia presentation, wherein the 
XML-based visual cue element is nested within the XML-based element for the 
associated visual component". 

Bugaj teaches a layout section of a SMIL document including alternative layout 
elements embedded in a switch element used to determine the placement of the 
presentation (section 5, General Semantics, lines 1-6), however, Bugaj is silent on 
wherein the XML- based visual cue element includes attributes that define temporal and 
spatial relativity between a display of the visual cue and a display of the multimedia 
component; and displaying the synchronized multimedia presentation including the 
visual cue superimposed over the multimedia component in a temporal and spatial 
relationship defined by the attributes of the XML-based visual cue element. 

The examiner maintains that it was well known in the art to provide "wherein the 
XML-based visual cue element includes attributes that define temporal and spatial 
relativity between a display of the visual cote and a display of the multimedia 
component; and displaying the synchronized multimedia presentation including the 
visual cue superimposed over the multimedia component in a temporal and spatial 
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relationship defined by the attributes of the XML-based visual cue element", as taught 
by Lemmons and Kaiser. 

In a similar field of endeavor Lemmons teaches wherein the XML-based visual 
cue element includes attributes that define temporal and spatial relativity between a 
display of the visual cote and a display of the multimedia component; and displaying the 
synchronized multimedia presentation including the visual cue superimposed together 
with the multimedia component in a temporal and spatial relationship defined by the 
attributes of the XML-based visual cue element (col. 9, II. 16-22, & figure 6B , & col. 8, II. 
58-67, & col. 9, II. 1-34, & figure 7A). 

Therefore, it would have been obvious to one of ordinary skill in the art at the 
time the invention was made to modify Bugaj by specifically providing "wherein the 
XML-based visual cue element includes attributes that define temporal and spatial 
relativity between a display of the visual cote and a display of the multimedia 
component; and displaying the synchronized multimedia presentation including the 
visual cue superimposed together with the multimedia component in a temporal and 
spatial relationship defined by the attributes of the XML-based visual cue element", as 
taught by Lemmons, for the purpose of providing interactive presentations wherein 
additional information on the presented media is available to user during the 
presentation. 

Bugaj and Lemmons are silent on displaying the synchronized multimedia 
presentation including the visual cue superimposed over the multimedia component. 
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Kaiser teaclies displaying the synchronized multinnedia presentation including the 
visual cue superimposed over the multimedia component (figure 6D, label 6300, & 
6600, & col. 11,11.20-26). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to modify the system of Bugaj and Lemmons to include 
displaying the synchronized multimedia presentation including the visual cue 
superimposed over the multimedia component as taught by Kaiser in order to allow 
users to easily recognize the association between the interactive icon and the displayed 
video, thereby increasing the viewer's convenience (col. 1, II. 49-55). 

Regarding claims 23 and 27, Bugaj is silent on a visual element attribute that 
defines a visual representation of the visual cue; a spatial element attribute that defines 
spatial characteristics of the visual cue; and a temporal element attribute that defines 
temporal characteristics of the visual cue. 

The examiner maintains that it was well known in the art to provide "a visual 
element attribute that defines a visual representation of the visual cue; a spatial element 
attribute that defines spatial characteristics of the visual cue; and a temporal element 
attribute that defines temporal characteristics of the visual cue", as taught by Lemmons. 

In a similar field of endeavor Lemmons teaches a visual element attribute that 
defines a visual representation of the visual cue; a spatial element attribute that defines 
spatial characteristics of the visual cue (col. 9, II. 16-22); and a temporal element 
attribute that defines temporal characteristics of the visual cue (figure 68 (because the 
elements a synchronized temporal element attribute is inherent)). 
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Therefore, it would have been obvious to one of ordinary sl<ill in the art at the 
time the invention was made to modify Bugaj by specifically providing "a visual element 
attribute that defines a visual representation of the visual cue; a spatial element attribute 
that defines spatial characteristics of the visual cue; and a temporal element attribute 
that defines temporal characteristics of the visual cue", as taught by Lemmons, for the 
purpose of providing interactive presentations wherein additional information on the 
presented media is available to user during the presentation. 

Regarding Claims 24 and 28, Bugaj teaches schedule elements with begin and 
end time wherein the duration is the difference between the end time and begin time of 
an element (section 6.1, General Semantics, line 3, & section 6.2 page 8, line 3) which 
meets "a method, wherein the temporal characteristics include at least two of begin 
time, end time, and duration". 

Regarding Claims 25 and 29, Bugaj teaches an image element tagged as an 
XML media object element (section 6.4, Syntax) which inherently meets "a method, 
wherein the visual representation includes at least one of a shape and a color of the 
visual cue". 

Claims 30-33 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Lemmons, in view of Kaiser. 

Regarding claim 30, Lemmons teaches an apparatus (figures 2-3) comprising: 
a display screen (36); a computer-readable storage medium for storing computer- 
executable process steps that cause a synchronized multimedia presentation to be 
displayed on the display screen (figure 6B, figure 3, label 42 (the circuitry meets 
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"storage medium and processor")), and for storing XML-based data for synchronizing 
tlie display of the multimedia presentation; and a processor to execute the process 
steps stored in the storage medium (col. 7, II. 5-6, & 20-23); wherein the process steps 
comprise: receiving the XML-based data, wherein the XML-based data includes an 
XML-based element for a visual cue together with an XML-based element for a visual 
component contained in the multimedia presentation, wherein the XML-based visual 
cue element is nested within the XML-based element for the associated visual 
component, and wherein the XML-based visual cue element includes attributes that 
define temporal and spatial relativity between a display of the visual cue and a display 
of the visual component; and displaying the synchronized multimedia presentation 
including the visual cue superimposed together with the visual component in a temporal 
and spatial relationship defined by the attributes of the XMIL-based visual cue element 
(figure 7A, entire, & figure 8, entire). 

Lemmons is silent on displaying the synchronized multimedia presentation 
including the visual cue superimposed over the visual component. 

Kaiser teaches displaying the synchronized multimedia presentation including the 
visual cue superimposed over the visual component (figure 6D, label 6300, & 6600, & 
col. 11, II. 20-26). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
the invention was made to modify the system of Lemmons to include displaying the 
synchronized multimedia presentation including the visual cue superimposed over the 
visual component as taught by Kaiser in order to allow users to easily recognize the 



Application/Control Number: 10/637,625 Page 18 

Art Unit: 2423 

association between tlie interactive icon and the displayed video, thereby increasing the 
viewer's convenience (col. 1, II. 49-55). 

Regarding claim 31 , Lemmons teaches a visual element attribute that defines a 
visual representation of the visual cue; a spatial element attribute that defines spatial 
characteristics of the visual cue (col. 9, II. 16-22); and a temporal element attribute that 
defines temporal characteristics of the visual cue (figure 6B (because the elements a 
synchronized temporal element attribute is inherent)). 

Regarding claim 32, Lemmons inherently teaches an apparatus wherein the 
temporal characteristics include at least two of begin time, end time, and duration (figure 
6A, entire, & figure 7A, entire (because synchronization is involved there has to be 
control points (for example start time end time, duration)). 

Regarding claim 33, Lemmons teaches an apparatus, wherein the visual 
representation includes at least one of a shape and a color of the visual cue (col. 9, II. 
16-22). 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to ANTHONY BANTAMOl whose telephone number is 
(571 )270-3581 . The examiner can normally be reached on Monday - Friday 8-5:30. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Andrew Koenig can be reached on (571) 272 7296. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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